Building QGIS from sources is not hard at all on a recent linux box, but what about if you wanted to be able to step-debug into Qt core or if you wanted to build QGIS agains the latest Qt release? Here things become tricky. This short post is about my experiments to build Qt and and other Qt-based dependencies for QGIS in order to get a complete debugger-friendly build of QGIS.   Start with downloading the latest Qt installer from Qt official website: https://www.qt.io/download-qt-for-application-development choose the Open Source version.   Now install the Qt version you want to build, make sure you check the Sources and the components you might need. Whe you are done with that, you’ll have your sources in a location like /home/user/Qt/5.9.3/Src/ To build the sources, you can change into that directory and issue the following command – I assume that you have already installed all the dependencies normally needed to build C++ Qt programs – I’m using clang here but feel free to choose gcc, we are going to install the new Qt build into /opt/qt593.

./configure -prefix /opt/qt593 -debug -opensource -confirm-license -ccache -platform linux-clang

When done, you can build it with

make -j9
sudo make install

  To build QGIS you also need three additional Qt packages   QtWebKit from https://github.com/qt/qtwebkit (you can just download the zip): Extract it somewhere and build it with

/opt/qt593/bin/qmake WebKit.pro
make -j9
sudo make install

  Same with QScintila2 from https://www.riverbankcomputing.com/software/qscintilla

/opt/qt593/bin/qmake qscintilla.pro
make -j9
sudo make install

  QWT is also needed and it can be downloaded from https://sourceforge.net/projects/qwt/files/qwt/6.1.3/ but it requires a small edit in qwtconfig.pri before you can build it: set QWT_INSTALL_PREFIX = /opt/qt593_libs/qwt-6.1.3 to install it in a different folder than the default one (that would possibly overwrite a system install of QWT). The build it with:

/opt/qt593/bin/qmake qwt.pro
make -j9
sudo make install

  If everything went fine, you can now configure Qt Creator to use this new debug build of Qt: start with creating a new kit (you can probably clone a working Qt5 kit if you have one). What you need to change is the Qt version (the path to cmake) to point to your brand new Qt build,: Pick up a name and choose the Qt version, but before doing that you need to click on Manage… to create a new one: Now you should be able to build QGIS using your new Qt build, just make sure you disable the bindings in the CMake configuration: unfortunately you’d also need to build PyQt in order to create the bindings.   Whe QGIS is built using this debug-enabled Qt, you will be able to step-debug into Qt core libraries! Happy debugging!  

The post Building QGIS master with Qt 5.9.3 debug build first appeared on Open Web Solutions, GIS & Python Development.

Whatever is your level of Python knowledge, when you’ll discover the advantages and super-powers of IPython you will never run the default python console again, really: never!

If you’ve never heard about IPython, discover it on IPython official website, don’t get confused by its notebook, graphics and parallel computing capabilities, it also worth if only used as a substitute for the standard Python shell.

I discovered IPython more than 5 years ago and it literally changed my life: I use it also for debugging instead ofpdb, you can embed an IPython console in your code with:

from IPython import embed; embed()

TAB completion with full introspection

What I like the most in IPython is its TAB completion features, it’s not just like normal text matching while you type but it has full realtime introspection, you only see what you have access to, being it a method of an instance or a class or a property, a module, a submodule or whatever you might think of: it even works when you’re importing something or you are typing a path like in open('/home/.....

Its TAB completion is so powerful that you can even use shell commands from within the IPython interpreter!

Full documentation is just a question mark away

Just type “?” after a method of function to print its docstring or its signature in case of SIP bindings.

Lot of special functions

IPython special functions are available for history, paste, run, include and many more topics, they are prefixed with “%” and self-documented in the shell.

All that sounds great! But what has to do with QGIS?

I personally find the QGIS python console lacks some important features, expecially with the autocompletion (autosuggest). What’s the purpose of having autocompletion when most of the times you just get a traceback because the method the autocompleter proposed you is that of another class? My brain is too small and too old to keep the whole API docs in my mind, autocompletion is useful when it’s intelligent enough to tell between methods and properties of the instance/class on which you’re operating.

Another problem is that the API is very far from being “pythonic” (this isn’t anyone’s fault, it’s just how SIP works), here’s an example (suppose we want the SRID of the first layer):

core.QgsMapLayerRegistry.instance().mapLayers().value()[0].crs().authid()
# TAB completion stops working here^

TAB completion stop working at the first parenthesis

What if all those getter would be properties?

registry = core.QgsMapLayerRegistry.instance()
# With a couple of TABs without having to remember any method or function name!
registry.p_mapLayers.values()
[<qgis._core.QgsRasterLayer at 0x7f07dff8e2b0>,
 <qgis._core.QgsRasterLayer at 0x7f07dff8ef28>,
 <qgis._core.QgsVectorLayer at 0x7f07dff48c30>,
 <qgis._core.QgsVectorLayer at 0x7f07dff8e478>,
 <qgis._core.QgsVectorLayer at 0x7f07dff489d0>,
 <qgis._core.QgsVectorLayer at 0x7f07dff48770>]

layer = registry.p_mapLayers.values()[0]

layer.p_c ---> TAB!
layer.p_cacheImage            layer.p_children       layer.p_connect       
layer.p_capabilitiesString    layer.p_commitChanges  layer.p_crs           
layer.p_changeAttributeValue  layer.p_commitErrors   layer.p_customProperty

layer.p_crs.p_ ---> TAB!
layer.p_crs.p_authid               layer.p_crs.p_postgisSrid      
layer.p_crs.p_axisInverted         layer.p_crs.p_projectionAcronym
layer.p_crs.p_description          layer.p_crs.p_recentProjections
layer.p_crs.p_ellipsoidAcronym     layer.p_crs.p_srsid            
layer.p_crs.p_findMatchingProj     layer.p_crs.p_syncDb           
layer.p_crs.p_geographicCRSAuthId  layer.p_crs.p_toProj4          
layer.p_crs.p_geographicFlag       layer.p_crs.p_toWkt            
layer.p_crs.p_isValid              layer.p_crs.p_validationHint   
layer.p_crs.p_mapUnits    

layer.p_crs.p_authid
Out[]: u'EPSG:4326'

This works with a quick and dirty hack: propertize that adds a p_... property to all methods in a module or in a class that

1. do return something
2. do not take any argument (except self)

this leaves the original methods untouched (in case they were overloaded!) still allowing full introspection and TAB completion with a pythonic interface.

IPyConsole: a QGIS IPython plugin

If you’ve been reading up to this point you probably can’t wait to start using IPython inside your beloved QGIS (if that’s not the case, please keep reading the previous paragraphs carefully until your appetite is grown!).

An experimental plugin that brings the magic of IPython to QGIS is now available:
Download IPyConsole

 

Please start exploring QGIS objects and classes and give me some feedback!

 

Installation notes

You basically need only a working IPython installation, IPython is available for all major platforms and distributions, please refer to the official documentation.

 

 

Today it’s a great day for QGIS Server: Python plugins, the project that took me busy during the past two months, has been merged to master and will be available starting with the next QGIS release.

The project has been discussed and approved in Essen during the last QGIS HF (see my presentation about server plugins), thanks to the input and suggestions coming from Marco Hugentobler and Martin Dobias it is now implemented in the more complete and flexible way.

In this article I will introduce the core concepts and the main features of python plugins for QGIS server.

QGIS server plugins architecture

QGIS server provides some core services: WFS, WMS, WCS. What we wanted to achieve was a system to easily add new services and modify existing services through python plugins.

Mi first experiments were limited to a 404 handler that intercepts unhandled requests and hooks into python plugins capturing every stdout output, this was indeed not enough flexible for a full fledged plugins implementation.

The main loop

QGIS server is not different from most web services implementations: it listens for incoming requests, parses the URL query string parameters and returns its output accordingly to the incoming request.

The standard loop before introducing python plugins looked like the following:

• Get the request
  • create GET/POST/SOAP request handler
  • if SERVICE is WMS/WFS/WCS
    • create WMS/WFS/WCS server passing in request handler
      • call server’s executeRequest()
        • call request handler output method
  • else Exception

Plugins come into play

Server python plugins are loaded once when the FCGI application starts and they should register one or more QgsServerFilter (from this point, you might find useful a quick look to the server plugins API docs). Each filter should implement at least one of three callbacks (aka: hooks):

1. requestReady
2. sendResponse
3. responseComplete

All filters have access to the request/response object (QgsRequestHandler) and can manipulate all its properties (input/output) and can raise exceptions (while in a quite particular way as we’ll see below).

Here is a pseudo code showing how and when the filter’s callbacks are called:

• Get the request

  • create GET/POST/SOAP request handler
  • pass request to serverIface
  • call plugins requestReady filters
  • if there is not a response
    • if SERVICE is WMS/WFS/WCS
      • create WMS/WFS/WCS server
        • call server’s executeRequest and possibily call sendResponse plugin filters when streaming output or store the byte stream output and content type in the request handler
    • call plugins responseComplete filters

  • call plugins sendResponse filters

  • request handler output the response

requestReady

This is called when the request is ready: incoming URL and data have been parsed and before entering the core services (WMS, WFS etc.) switch, this is the point where you can manipulate the input and perform actions like:

• authentication/authorization
• redirects
• add/remove certain parameters (typenames for example)
• raise exceptions

You could even substitute a core service completely by changing SERVICE parameter and hence bypassing the core service completely (not that this make much sense though).

 

sendResponse

This is called whenever output is sent to FCGI stdout (and from there, to the client), this is normally done after core services have finished their process and after responseComplete hook was called, but in a few cases XML can become so huge that a streaming XML implementation was needed (WFS GetFeature is one of them), in this case, sendResponse is called multiple times before the response is complete (and before responseComplete is called). The obvious consequence is that sendResponse is normally called once but might be exceptionally called multiple times and in that case (and only in that case) it is also called before responseComplete.

SendResponse is the best place for direct manipulation of core service’s output and while responseComplete is typically also an option, sendResponse is the only viable option  in case of streaming services.

responseComplete

This is called once when core services (if hit) finish their process and the request is ready to be sent to the client. As discussed above, this is  normally called before sendResponse except for streaming services (or other plugin filters) that might have called sendResponse earlier.

responseComplete is the ideal place to provide new services implementation (WPS or custom services) and to perform direct manipulation of the output coming from core services (for example to add a watermark upon a WMS image).

Raising exception from a plugin

Some work has still to be done on this topic: the current implementation can distinguish between handled and unhandled exceptions by setting a QgsRequestHandler property to an instance of QgsMapServiceException, this way the main C++ code can catch handled python exceptions and ignore unhandled exceptions (or better: log them).

This approach basically works but it does not satisfy my pythonic way of handle exceptions: I would rather prefer to raise exceptions from python code to see them bubbling up into C++ loop for being handled there.

Conclusions

The new plugin system is very flexible and allows for basic input/output (i.e. request/response) manipulation and for new services implementation while it remains unobtrusive and has negligible impact on performances, in the next article I will discuss server plugin implementation in depth.

 

See also the second part of this article.

See all QGIS Server related posts